feat: add Uint8Array support for zero-copy worker_threads transfer by umeshmore45 · Pull Request #3183 · redis/node-redis

umeshmore45 · 2026-02-21T23:55:31Z

Description

This PR adds comprehensive Uint8Array support to enable zero-copy data transfer to worker_threads, significantly improving performance for multi-threaded applications handling large payloads.

Motivation

Currently, Redis values are returned as Buffer, which requires copying when transferring to worker threads. With Uint8Array support, the underlying ArrayBuffer can be transferred with zero-copy semantics.

Performance Benefits:

Zero memory copying when transferring to workers
Reduced memory usage for large payloads
Better performance in multi-threaded applications

For a 10MB payload, this eliminates 10MB of memory copying when transferring between threads.

Usage Example

import { createClient, RESP_TYPES } from 'redis';
import { Worker } from 'worker_threads';

const client = await createClient().connect();

// Get data as Uint8Array
const data = await client
  .withTypeMapping({
    [RESP_TYPES.BLOB_STRING]: Uint8Array
  })
  .get('large-file');

// Zero-copy transfer to worker
const worker = new Worker('./worker.js');
worker.postMessage(data, [data.buffer]);

// After transfer, data.byteLength === 0 (ownership transferred)

Changes Made

Type System (types.ts):

Extended BlobStringReply, SimpleStringReply, VerbatimStringReply to support Uint8Array
Updated UnwrapConstructor and ReplyWithTypeMapping to handle Uint8Array
Added Uint8Array to RedisArgument type

Decoder (decoder.ts):

Added Uint8Array conversion in 4 string decoding methods (single/multi-chunk)
Handles both simple strings and blob strings correctly
Maintains backward compatibility with Buffer

Encoder (encoder.ts):

Added Uint8Array handling in command encoding
Properly ordered Buffer check before Uint8Array (Buffer extends Uint8Array)
Updated error messages to include Uint8Array

Command Support:

Updated type guards in HSET, GEOSEARCH, SEARCH, AGGREGATE to accept Uint8Array arguments
Fixed Buffer/Uint8Array handling in cluster-slots.ts and time-series/helpers.ts

Testing:

Added 4 decoder tests (simple/blob/verbatim strings)
Added 1 encoder test
Added 1 integration test for type mapping
All existing tests continue to pass

Breaking Changes

None! This is fully backward compatible:

Buffer remains the default type
Existing code continues to work unchanged
Uint8Array is opt-in via withTypeMapping()

Files Changed

12 files modified
95 lines added, 19 lines removed
Net addition: 76 lines

Closes #3106

Checklist

Does npm test pass with this change (including linting)?
Is the new or changed code fully tested?
Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?

Note: Documentation can be added in a follow-up PR if needed. The code includes inline comments and the PR description provides usage examples.

Note

Medium Risk
Touches core request/response encoding/decoding, reply typing, and queue/caching invalidation paths; regressions could affect many commands and pub/sub/monitor behavior. Changes are largely additive and backed by targeted regression tests for Uint8Array mappings.

Overview
Adds first-class Uint8Array support end-to-end: RedisArgument and reply types now include Uint8Array, and the RESP decoder/encoder can emit/accept plain Uint8Array when configured via typeMapping.

Fixes a set of Uint8Array regressions caused by Buffer-only guards and Uint8Array#toString() semantics, including numeric parsing (GEODIST, ZRANK WITHSCORE), tuple/map key decoding (e.g. HGETALL, PUBSUB NUMSUB), cluster slot calculation, and RESP2 pub/sub/monitor/reset handling in commands-queue.

Updates client-side cache invalidation to normalize Redis keys consistently across string/Buffer/Uint8Array and adds comprehensive tests covering encoding/decoding, type-mapping, command argument parsing, and cache/queue behaviors.

^{Written by Cursor Bugbot for commit 64936ec. This will update automatically on new commits. Configure here.}

- Add Uint8Array to type system (BlobStringReply, SimpleStringReply, VerbatimStringReply) - Implement Uint8Array encoding/decoding in RESP protocol handlers - Support Uint8Array as command arguments across all commands - Add comprehensive test coverage for Uint8Array operations - Fix type guards in HSET, GEOSEARCH, AGGREGATE, SEARCH commands - Fix Buffer/Uint8Array handling in time-series helpers and cluster-slots - Maintain full backward compatibility with Buffer Enables zero-copy transfer to worker_threads via ArrayBuffer transfer. Closes redis#3106

jit-ci · 2026-02-21T23:55:48Z

Hi, I’m Jit, a friendly security platform designed to help developers build secure applications from day zero with an MVS (Minimal viable security) mindset.

In case there are security findings, they will be communicated to you as a comment inside the PR.

Hope you’ll enjoy using Jit.

Questions? Comments? Want to learn more? Get in touch with us.

nkaradzhov · 2026-02-23T09:17:39Z

Thanks @umeshmore45, i will take a look

nkaradzhov

Please address the below:

Uint8Array keys are misclassified in sorted-set key parsing (ZINTER/ZUNION/ZUNIONSTORE).
Add parseArgs regression cases in ZINTER.spec.ts, ZUNION.spec.ts, and ZUNIONSTORE.spec.ts for single and array Uint8Array keys.
CONFIG SET single-parameter form treats Uint8Array as object entries.
Add regression in CONFIG_SET.spec.ts verifying single-parameter Uint8Array input uses the correct branch.
CMS.MERGE has the same plain-argument discriminator bug for Uint8Array[].
Add regression in MERGE.spec.ts for plain Uint8Array[] sources (no weights).
Client-side cache invalidation key normalization is inconsistent for Uint8Array vs Buffer/string.
Add regression in cache.spec.ts proving a cached Uint8Array key is invalidated by equivalent server-style invalidation payloads.
Tuple/map key transforms use .toString() on Uint8Array, producing byte-list keys ("102,111,111").
Add regressions in HGETALL.spec.ts and PUBSUB_NUMSUB.spec.ts asserting semantic string keys are preserved under Uint8Array mapping.
Numeric transforms parse Uint8Array replies as NaN.
Add command-level regressions in GEODIST.spec.ts, ZRANK_WITHSCORE.spec.ts, and GET.spec.ts.
RESET handling in queue does not recognize Uint8Array SIMPLE_STRING replies and can hang in monitor/type-mapped flows.
Add a new queue unit test (under packages/client/lib/client, e.g. commands-queue.spec.ts) covering monitor with SIMPLE_STRING -> Uint8Array and ensuring reset() resolves.
RESP2 pub/sub PING resolution path ignores Uint8Array mapping behavior.
Add queue-level regression test proving RESP2 pub/sub PING respects SIMPLE_STRING mapping when mapped to Uint8Array.
Test strictness must be tightened so type regressions are detectable.
Replace permissive deepEqual with deepStrictEqual where binary type identity matters in decoder.spec.ts:66.
For all new Uint8Array assertions, require both instanceof Uint8Array and !Buffer.isBuffer(value) (including types.spec.ts:54).

For each item, include both the code fix and a regression test that would fail without the fix and pass with it.

…t8Array as valid key types - Updated type guard in generic-transformers to recognize Uint8Array. - Added unit tests for Uint8Array support in ZINTER, ZUNION, and ZUNIONSTORE commands. - Ensured compatibility with existing command structures while expanding argument handling. This change improves the flexibility of command arguments, allowing for zero-copy transfers with Uint8Array.

- Updated type guard to recognize Uint8Array as a valid parameter type. - Added a regression test to ensure correct handling of Uint8Array keys. - Improved argument parsing to prevent incorrect behavior with Uint8Array inputs. This change enhances the flexibility of the CONFIG SET command, aligning with recent updates for better Uint8Array support across commands.

- Updated type guard in isPlainSketches to include Uint8Array. - Added regression test to verify correct handling of Uint8Array sources in CMS.MERGE. This change improves the argument handling of the CMS.MERGE command, ensuring compatibility with Uint8Array inputs.

- Introduced tests to ensure proper invalidation of cache entries when using Uint8Array keys. - Validated that entries are evicted correctly when invalidation keys are provided as Buffer or string. - Enhanced the client-side cache functionality to handle Uint8Array keys consistently with server-side invalidation. This change improves the reliability of cache behavior in scenarios involving Uint8Array keys.

… commands - Introduced regression tests to ensure Uint8Array field keys and channel names are correctly decoded as UTF-8 strings. - Validated that the transformation of replies maintains the integrity of keys when using Uint8Array. This change enhances the reliability of key handling in the HGETALL and PUBSUB NUMSUB commands, aligning with recent updates for better Uint8Array support.

…, GET, and ZRANK_WITHSCORE commands - Introduced tests to validate that Uint8Array replies are correctly parsed as finite numbers instead of NaN. - Ensured that the transformation of replies handles Uint8Array correctly across multiple commands, addressing a coercion hazard. This change enhances the reliability of numeric handling in commands that utilize Uint8Array, improving overall data integrity.

- Refactored the handling of the reply to use a more robust check for Uint8Array. - Enhanced the condition to ensure accurate detection of replies, improving the reliability of the command queue's response handling. This change addresses potential issues with reply processing in PubSub and MONITOR modes.

…ray and Buffer - Added tests to ensure the PING command resolves correctly as Uint8Array, Buffer, or string based on the provided typeMapping. - Improved the handling of PONG replies to accurately return the expected types, enhancing the command queue's response reliability. This change addresses potential inconsistencies in the response types for the PING command, aligning with recent updates for better Uint8Array support.

- Changed to in decoder tests to ensure strict equality checks. - Enhanced type checks in types tests to clarify expectations for Uint8Array instances and prevent Buffer confusion. These changes improve the reliability of test assertions, ensuring more accurate validation of expected outcomes.

cursor

Cursor Bugbot has reviewed your changes and found 3 potential issues.

packages/client/lib/cluster/cluster-slots.ts

packages/client/lib/commands/GEODIST.spec.ts

packages/client/lib/client/cache.spec.ts

…rray and Buffer - Introduced a new utility function to standardize the conversion of Redis keys from string, Buffer, or Uint8Array to UTF-8 strings. - Updated cache management in to utilize the new key conversion function, ensuring consistent key handling. - Added function to convert Redis blob replies to UTF-8 strings, improving the handling of Uint8Array and Buffer replies across multiple commands. - Refactored commands such as GEODIST, PUBSUB_NUMSUB, and ZRANK_WITHSCORE to leverage for accurate reply transformations. These changes enhance the reliability and consistency of key and reply handling, particularly for Uint8Array and Buffer types, aligning with recent updates for better data integrity.

…ecoder and encoder - Cleaned up the code by removing unnecessary comments that indicated Uint8Array support in the decoder and encoder files. - This change improves code readability and maintains focus on the relevant logic without cluttering with repetitive comments.

umeshmore45 marked this pull request as ready for review February 21, 2026 23:57

umeshmore45 mentioned this pull request Feb 22, 2026

Support Uint8Array values for zero-copy transfer to worker_threads #3106

Open

nkaradzhov self-requested a review March 2, 2026 13:23

nkaradzhov requested changes Mar 2, 2026

View reviewed changes

umeshmore45 added 9 commits March 2, 2026 23:36

cursor bot reviewed Mar 3, 2026

View reviewed changes

packages/client/lib/cluster/cluster-slots.ts Outdated Show resolved Hide resolved

packages/client/lib/commands/GEODIST.spec.ts Show resolved Hide resolved

packages/client/lib/client/cache.spec.ts Show resolved Hide resolved

umeshmore45 added 2 commits March 3, 2026 13:56

umeshmore45 requested a review from nkaradzhov March 3, 2026 09:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Uint8Array support for zero-copy worker_threads transfer#3183

feat: add Uint8Array support for zero-copy worker_threads transfer#3183
umeshmore45 wants to merge 12 commits intoredis:masterfrom
umeshmore45:feature/uint8array-support

umeshmore45 commented Feb 21, 2026 •

edited by cursor bot

Loading

Uh oh!

jit-ci bot commented Feb 21, 2026

Uh oh!

nkaradzhov commented Feb 23, 2026

Uh oh!

nkaradzhov left a comment •

edited

Loading

Uh oh!

cursor bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

umeshmore45 commented Feb 21, 2026 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation

Usage Example

Changes Made

Breaking Changes

Files Changed

Checklist

Uh oh!

jit-ci bot commented Feb 21, 2026

Uh oh!

nkaradzhov commented Feb 23, 2026

Uh oh!

nkaradzhov left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

umeshmore45 commented Feb 21, 2026 •

edited by cursor bot

Loading

nkaradzhov left a comment •

edited

Loading